
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
■P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



| APPLICATION NO. j 


FILING DATE 


FIRST NAMED INVENTOR 


| ATTORNEY DOCKET NO. j 


CONFIRMATION NO. | 


10/708,905 


03/31/2004 


Ibrahim M. Mohamed 


H0005395 


2904 



128 7590 03/06/2008 

HONEYWELL INTERNATIONAL INC. 
101 COLUMBIA ROAD 
P O BOX 2245 

MORRISTOWN, NJ 07962-2245 



EXAMINER 



FINDLEY, CHRISTOPHER G 



ART UNIT 



2621 



PAPER NUMBER 



MAIL DATE 



DELIVERY MODE 



03/06/2008 



PAPER 



Please find below and/or attached an Office communication concerning this application or proceeding. 



The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



* 

Office Action Summary 


Application No. 

10/708,905 


Appiicant(s) 

MOHAMEDETAL 


Examiner 

kaAH 1 1 1 1 1 IVI 

CHRISTOPHER FINDLEY 


Art Unit 

2621 





~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

I) 13 Responsive to communication(s) filed on 12 December 2007 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 7-28 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) EE3 Claim(s) U28 is/are rejected. 

7) \Z\ Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

I I) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 ,□ Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 

30 Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) □ Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date . 



4) O Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) O Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20080228 



Application/Control Number: 10/708,905 Page 2 

Art Unit: 2621 

DETAILED ACTION 
Response to Arguments 

1 . Applicant's arguments filed 12/12/2007 have been fully considered but they are 
not persuasive. 

2. Re claims 1,16, and 23, the Applicant contends that the cited prior art (Toklu) 
neither teaches nor suggests that a rate of change of visual content is determined with 
the rate of change representing a difference of a first value and a second value, said 
first value representing a change of visual content of a current frame compared to a first 
frame, said second value representing a change of visual content of said first frame 
compared to a second frame, wherein said second frame is a reference frame for said 
first frame and said first frame is a reference frame for said current frame. However, the 
Examiner respectfully disagrees. Toklu discloses calculating an amount of motion 
between two frames (Toklu: column 8, line 15, through column 9, line 9). The motion 
activity found is then used to generate a motion activity (MA) curve (Toklu: column 9, 
lines 13-31 and Eqn. 7). Once the possible key frames are found through the process 
of Figs. 2A-2C in Toklu, the potential key frames undergo an elimination process (Toklu: 
Fig. 3), wherein the histograms for candidate frames are compared with the histogram 
for a reference frame (Toklu: column 13, lines 14-29). The motion activity for the 
candidate frame represents a first value, and the reference frame represents a second 
value, which are compared for the purpose of determining key frames. 

3. Re claim 2, the Applicant contends that the cited prior art fails to teach or suggest 
that the rate is computed as a difference of a first representative magnitude and a 
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second representative magnitude. However, the Examiner respectfully disagrees. 
Upon further review of the prior art, Toklu discloses in Eqn. 7 that the MA value is 
calculated by taking a square root of a squared value, which by definition yields an 
absolute value, or magnitude, for each (x, y) coordinate entered into the equation. 

4. Re claim 6, the Applicant contends that the cited prior art fails to teach or suggest 
that only active pixels are used in computing the displacement magnitude. However, 
the Examiner respectfully disagrees. One of ordinary skill in the art at the time of the 
invention would have found it obvious that if a pixel were inactive (meaning that no 
change was present), there would be no zoom, rotation, pan or tilt values to be entered 
into Eqn. 7 of Toklu, and that pixel would not add to the motion activity (MA) value. 

5. Therefore, the Examiner maintains the previous rejection of claims 1-28. A 
modified copy of the previous rejection, reflecting changes made via the amendment 
filed 12/12/2007, is included below. 

Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed 
publication in this or a foreign country, before the invention thereof by the applicant for a patent. 

2. Claims 1-2, 15-17, and 23-24 are rejected under 35 U.S.C. 102(a) as being 
anticipated by Toklu et al. (US 6549643 B1). 
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Re claim 1, Tolku discloses a method of identifying key video frames in a 

* 

sequence of image frames, each of said sequence of image frames containing a 
plurality of pixels, each of said plurality of pixels corresponding to a corresponding point 
of an area based on which said sequence of image frames are generated, said method 
comprising: determining a rate of change of visual content of each current frame from a 
corresponding reference frame, each of said current frame and said reference frame 
being comprised in said sequence of image frames (Toklu: Fig. 2B, step 212), wherein 
said rate of change represents a difference of a first value and a second value, said first 
value representing a change of visual content of a current frame compared to a first 

i 

frame, said second value representing a change of visual content of said first frame 
compared to a second frame, wherein said second frame is a reference frame for said 
first frame and said first frame is a reference frame for said current frame (Toklu: 
column 8, line 15, through column 9, line 9, calculating an amount of motion between 
two frames; Toklu: column 9, lines 13-31 and Eqn. 7, motion activity found is then used 
to generate a motion activity (MA) curve; Toklu: Fig. 3, the potential key frames undergo 
an elimination process; Toklu: column 13, lines 14-29, the histograms for candidate 
frames are compared with the histogram for a reference frame, wherein the motion 
activity for the candidate frame represents a first value, and the reference frame 
represents a second value, which are compared for the purpose of determining key 
frames); and selecting said current frame as a key video frame (Toklu: Fig. 2B, step 
21 8)if said rate exceeds a first threshold value (Toklu: Fig. 2B, step 214). 
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Re claim 2, Tolku discloses a majority of the features of claim 2, as discussed 
above in claim 1, and Toklu additionally discloses determining a displacement 
magnitude of each moved pixel of said current frame compared to the position in said 
first frame and of said first frame compared to the position in said second frame (Toklu: 
column 8, line 15, through column 9, line 9, calculating an amount of motion between 
two frames); and computing a first representative magnitude of said displacement 
magnitude for said moved pixels of said current frame compared to said first frame, and 
a second representative magnitude of said displacement magnitude for said moved 
pixels of said first frame compared to said second frame (Toklu: column 8, line 15, 
through column 9, line 9, calculating an amount of motion between two frames; Toklu: 
column 9, lines 13-31 and Eqn. 7, motion activity found is then used to generate a 
motion activity (MA) curve), wherein said first value and said second value respectively 
equal said first representative magnitude and said second representative magnitude 
such that said rate is computed as a difference of said first representative magnitude 
and said second representative magnitude (Toklu: Fig. 3, the potential key frames 
undergo an elimination process; Toklu: column 13, lines 14-29, the histograms for 
candidate frames are compared with the histogram for a reference frame, wherein the 
motion activity for the candidate frame represents a first value, and the reference frame 
represents a second value, which are compared for the purpose of determining key 
frames). 
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Re claim 1 5, Tolku discloses that said first frame and said current frame are 
respectively adjacent to said second frame and said first frame in said sequence of 
frames (Toklu: column 5, line 65, through column 6, line 7). 

Claim 16 recites the corresponding computer readable medium containing a 
computer program for executing the method of claim 1 , and, therefore, has been 
analyzed and rejected with respect to claim 1 above. 

Claim 17 has been analyzed and rejected with respect to claim 2 above. 

Claim 23 recites the corresponding system for executing the method of claim 1, 
and, therefore, has been analyzed and rejected with respect to claim 1 above. 

Claim 24 has been analyzed and rejected with respect to claim 2 above. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall hot be negatived by the manner in which the invention was made. 

4. Claims 3-4, 6, 9-11, 13, 18-19, 21, 25-26, and 28 are rejected under 35 U.S.C. 

< 

103(a) as being unpatentable over Tolku et al. (US 6549643 B1) in view of Zhang et 
al. (US 7027513 B2). 

Re claim 3, Toklu discloses a majority of the features of claim 3, as discussed 
above in claim 2, but Toklu does not explicitly state that said first representative 
magnitude for said current frame equals an average of motion energy vector 
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magnitudes of said moved pixels of said current frame in comparison with 
corresponding pixels of said first frame. However, Zhang discloses a method for 
extracting key frames from video using a triangle model of motion based on perceived 
motion energy, where the average magnitude of the motion vectors for a particular 
frame is calculated (Zhang: column 9, lines 25-38), and the average magnitude is used 
to calculate the perceived motion energy (Zhang: column 9, lines 39-49). Since both 
Tolku and Zhang relate to selecting key video frames based on motion analysis, one of 
ordinary skill in the art at the time of the invention would have found it obvious to 
combine the triangle method of Zhang with the histogram analysis of Tolku in order to 
provide a uniform and consistent selection method that yields a proper number of key 
frames that are most representative of the video sequence content (Zhang: column 5, 
lines 34-37). The combined method of Tolku and Zhang has all of the features of claim 
3. 

Re claim 4, the combined method of Tolku and Zhang discloses that said current 
frame is selected as said key video frame only if said first representative magnitude 
exceeds a second threshold (Toklu: Fig. 2B, when the cumulative motion in steps 220, 
221 , and 222 exceeds a threshold, the frame is selected as a key frame in step 223). 

Re claim 6, the combined method of Tolku and Zhang discloses identifying a 

■ 

plurality of active pixels in said current frame, wherein a pixel is considered an active 
pixel if a corresponding displacement magnitude is outside of a range, wherein only said 
plurality of active pixels are used by said computing (Toklu: Fig. 2C, step 231). 
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Re claim 9, the combined method of Tolku and Zhang discloses enabling a user 
to specify one of a plurality of key video frames, wherein said plurality of key video 
frames are selected by said selecting (Zhang: Fig. 1, element 120; column 5, lines 43- 
44, indicate that the user may use the key frames to select the desired section of the 
video for display); and displaying said specified one of said plurality of key video frames 
(Zhang: Fig. 1, element 120; column 5/lines 43-44, indicate that the user may use the 
key frames to select the desired section of the video for display). 

Re claim 10, the combined method of Tolku and Zhang discloses displaying a 
prior key video frame and a next key video frame in relation to said specified one of said 
plurality of key video frames, wherein said prior key video frame and said next key video 
frame are comprised in said plurality of key video frames (Zhang: Fig. 1, element 120, 
key frames are displayed in a sequence). 

Re claim 11, the combined method of Tolku and Zhang discloses generating a 
display indicating the manner in which said plurality of key video frames are 
interspersed in said sequence of image frames, wherein said enabling is based on said 
display (Zhang: Fig. 12, the key frames are shown along with their corresponding frame 
numbers). 

Re claim 13, the combined method of Tolku and Zhang discloses generating a 
display listing said plurality of key video frames, wherein said enabling is based on said 
display (Zhang: Fig. 1, element 120; column 5, lines 43-44). 

Claim 18 has been analyzed and rejected with respect to claim 3 above. 

Claim 19 has been analyzed and rejected with respect to claim 4 above. 
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Claim 21 has been analyzed and rejected with respect to claim 6 above. 
Claim 25 has been analyzed and rejected with respect to claim 3 above. 
Claim 26 has been analyzed and rejected with respect to claim 4 above. 
Claim 28 has been analyzed and rejected with respect to claim 6 above. 

5. Claims 5, 7, 8, 20, 22, and 27 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Tolku et al. (US 6549643 B1) in view of Zhang et al. (US 
7027513 B2) as applied to claims 3-4, 6, 9-11, 13, 18-19, 21, 25-26, and 28 above, 
and further in view of Ma et al. (US 20040088723 A1 ). 

Re claim 5, the combined method of Tolku and Zhang discloses a majority of the 
features of claim 5, as discussed above in claims 1-4, but does not specifically disclose 
that the first threshold and the second threshold are adjusted dynamically to ensure that 
a desired number of frames are selected as key video frames in a specified duration. 
However, Ma discloses a method for generating a video summary, where a binarization 

* 

threshold is estimated in an adaptive manner (Ma: paragraph [0081]) when analyzing a 

video sequence for selecting key frames. Since Tolku, Zhang, and Ma relate to 

» 

selecting key frames in a video sequence, one of ordinary skill in the art at the time of 
the invention would have found it obvious to combine the attention modeling of Ma with 
the combined key frame selection method of Tolku and Zhang in order to create a 
representative video summary consisting of very short video clips, which contain the 
video immediately preceding and immediately following key frames that have been 
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selected (Ma: Fig. 20). The combined method of Tolku, Zhang, and Ma has all of the 
features of claim 5: 

4 

Re claim 7, the combined method of Tolku, Zhang, and Ma discloses that said 
range set by a distance of two times the variance from the mean of a distribution (Ma: 
paragraphs [0081]-[0082], the threshold is a function of the variance). 

Re claim 8, the combined method of Tolku, Zhang, and Ma discloses that said 
representative magnitude comprises an average of said active pixels (Ma: equation (12) 
and paragraph [0084]). 

Claim 20 has been analyzed and rejected with respect to claim 5 above. 

Claim 22 has been analyzed and rejected with respect to claim 7 above. 

Claim 27 has been analyzed and rejected with respect to claim 5 above. 

6. Claims 12 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Tolku et al. (US 6549643 B1) in view of Zhang et al. (US 7027513 B2) as 
applied to claims 3-4, 6, 9-11, 13, 18-19, 21, 25-26, and 28 above, and further in 
view of Sull et al. (US 2006006471 6 A1 ). 

Re claim 12, the combined method of Tolku and Zhang discloses a majority of 
the features of claim 12, as discussed above in claims 1-4, 6, and 9-11, but does not 
specifically disclose that the display comprises a pie chart. However, Sull discloses 
techniques for navigating multiple video streams, where textual/visual information, such 
as a pie chart, may be displayed along with poster-thumbnails on the video selection 
screen for the user interface (Sull: paragraph [0307]). Since Tolku, Zhang, and Sull all 
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relate to representative images for video sequences, one of ordinary skill in the art at 
the time of the invention would have found it obvious to combine the textual information 
of Sull with the key frame selection of the combined method of Tolku and Zhang in order 
to provide the user with more information, such as date and time of broadcast (Sull: 
paragraph [0307]), for improving the user's ability to quickly find the desired video 
segment (Zhang: column 5, lines 44-45). The combined method of Tolku, Zhang, and 
Sull has all of the features of claim 12. 

Re claim 14, the combined method of Tolku, Zhang, and Sull discloses that said 
display comprises a button, which when selected, causes said display to be generated 
(Sull: Figs. 6A and 6B, the thumbnail images act as buttons, so that when selected by 
the user, the corresponding video clip is displayed). 

Conclusion 

7. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure: 

a. Feature based hierarchical video segmentation 
Bozdagi et al. (US 6493042 B1) 

b. Method of selecting key-frames from a video sequence 
Wilfetal. (US 7184100 B1) 

c. System for automatic video segmentation and key frame extraction for 

* 

video sequences having both sharp and gradual transitions 
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Zhang et al. (US 5635982 A) 
8. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Contact 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to CHRISTOPHER FINDLEY whose telephone number is 
(571)270-1 199. The examiner can normally be reached on Monday-Friday 8:30am- 
5pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Marsha D. Banks-Harold can be reached on (571) 272-7905. The fax 
phone number for the organization where this application or proceeding is assigned is 
571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Christopher Findley/ 
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